#在2个文件中存放了英文计算机技术文章(可以选择2篇关于Python技术文件操作处理技巧的2篇英文技术文章), 请读取文章内容,进行词频的统计;并分别输出统计结果到另外的文件存放;
#比较这2篇文章的相似度(如果词频最高的前10个词,重复了5个,相似度就是50%;重复了6个,相似度就是60% ,......);
import re, collections

pattern = r',|\.|/|;|\'|`|\[|\]|<|>|\?|:|"|\{|\}|\~|!|@|#|\$|%|\^|&|\(|\)|-|=|\_|\+|，|。|、|；|‘|’|【|】|·|！| |…|（|）'
similarity_num = 0

with open( '1.txt', 'r' ) as f1:

    with open( '2.txt', 'r' ) as f2:

        text1 = f1.read()
        text2 = f2.read()

        words1 = re.split( pattern, text1 )
        words2 = re.split( pattern, text2 )

        words1_dict = collections.Counter( words1 )
        words2_dict = collections.Counter( words2 )

special_char = [ '', '\n', ' ', '\n\n', '\n' * 3, '\n' * 4, '\n' * 5, '\n' * 6 ]

words1_sorted = sorted( words1_dict.items(), key = lambda item : item[1], reverse = True )
words2_sorted = sorted( words2_dict.items(), key = lambda item : item[1], reverse = True )

top5_words1 = []
top5_words2 = []

with open( 'word1.txt', 'w' ) as f1:
    f1.write( 'Word\t\tFrequency\n' )
    for word1 in words1_sorted:
        if word1[0] not in special_char:
            f1.write( str( word1[0] ) + '\t\t\t' + str( word1[1] ) + '\n' )

with open( 'word2.txt', 'w' ) as f2:
    f2.write( 'Word\t\tFrequency\n' )
    for word2 in words2_sorted:
        if word2[0] not in special_char:
            f2.write( str( word2[0] ) + '\t\t\t' + str( word2[1] ) + '\n' )

with open( 'word1.txt', 'r' ) as f1:
    f1.readline()
    for line in f1.readlines():
        top5_words1.append( line.strip().split( '\t\t\t' ) )
        if len( top5_words1 ) >= 10:
            break

with open( 'word2.txt', 'r' ) as f2:
    f2.readline()
    for line in f2.readlines():
        top5_words2.append( line.strip().split( '\t\t\t' ) )
        if len( top5_words2 ) >= 10:
            break

for word1 in top5_words1:
    for word2 in top5_words2:
        if word1[0] in word2:
            similarity_num += 1

print( top5_words1 )
print( top5_words2 )
print( 'Similarity: {:.0%}'.format( similarity_num / 10 ) )